Bias

• Most scientific enterprises attempt to estimate some quantity or effect
• Bias is present if our estimate of that effect systematically deviates from the true value
• When thinking about bias, it is important to clearly define the estimand: What is it that we are trying to estimate?
  • \(bias = estimand - estimate\)

Estimate the right thing

A well described estimand is characterized by

• a clear effect measure
  • i.e., not just “Stress”, but “Stress, operationalized as …”
• a clear target population
  • for which set of humans do we want to measure the effect?

Internal and external validity

• If a study has internal validity, we correctly estimate the effect of a manipulation in the study sample
• If a study has external validity, the estimated effect in our study sample is the same as in the target population

• Studies with either internal or external validity can still be biased
• Internal and external validity alone are necessary but not sufficient conditions for obtaining unbiased estimates.

Bias in observational studies

Nobody needs convincing that confounding is a problem

Treatment randomisation prevent confounding.

Selection bias

If the treatment determines who stays in the study and there is another variable L that causes who stays and the outcome, there will be selection bias

Selection bias in obs. & exp. studies

Experiments with drop out also require the assumption of no unobserved confounding!

Bias in experimental studies I

If there are some characteristics C that influence the effect of the treatment T on the outcome O (T-C-interaction), and the distribution of C is different in the study sample and target population, a randomised experiment will produce biased effect estimates.

“Bias” in experimental studies II

Translational research needs experiments and observation

• Results from observational studies can be biased
• But the same is true for experiments
• Experiments can be improved through random sampling of participants and investigation of multiple outcomes
• Even then, many experiments cannot be performed for ethical reasons

Does parental income influence children’s mental health?

Does parental income influence children’s mental health?

Study designs

We analyse one data set to see in how far the results of Sariaslan and Kinge depend on modelling assumptions.
When the outcome is binary, sibling designs and panel designs with FE can only use a subset of data: Siblings and individuals with discordant outcomes, respectively.

Samples sizes and prevalence

Choosing a particular study design introduces substantial differences between the study sample and the source and target population.

Results: Odds & hazard ratios

Association is stronger when parents do not already have a diagnosis.

Results: Odds & hazard ratios

Original results of Kinge at al (2021) and Sariaslan et al (2021) can be replicated in this data set.

Results: Odds & hazard ratios

Only a logistic (or cox) regression with adjustment finds an association.

Summary of results

• We can replicate the results of Kinge et al. (there is an association) and Sariaslan et al. (there is no association) using the same basic data set
• Conditional logit regression also shows no effect

Sibling design and fixed effects panel model make fewer assumptions / have a higher internal validity. Should we rather trust these results?

Income variability

Income variation in designs with high internal validity is only a fraction (~20%) of the income variability between families.

Selection bias

If parental income is a cause of # children & we are looking for a negative effect of income, sibling designs show a biased (attenuated) association. Sibling design studies have higher internal validity lower but external validity.

Summary

• Different results of Kinge et al. (2021) and Sariaslan et al. (2021) are likely due to the choice of different analysis approaches
• There is no categorical difference between “non-causal” and “causal” designs.
• Causal inference always needs assumptions, which should be scrutinized.